Image recognition method and apparatus for the same method

ABSTRACT

An image recognizing method is provided, which includes the capability of efficiently recognizing characters such as letters and numerical characters included in an original image with accuracy when at least one of the characters is composed of plural elements. In this method, the elements in the original image are extracted to obtain a second image, in which each of the elements is enclosed by a rectangular frame. Then, a composite image is prepared with respect to a combination of the rectangular frames in the second image. After calculating a characteristic amount of the elements included in the composite image, the characteristic amount is input in a back-propagation network, in which learning about a reference character(s) to be included in the image has already finished, so that a degree of agreement between the characteristic amount of the composite image and the reference character is obtained. The composite image having a highest degree of agreement therebetween is determined from results provided by the back-propagation network with respect to different combinations of the rectangular frames in the second image, to output it as a recognition data.

TECHNICAL FIELD

[0001] The present invention relates to an image recognition method forefficiently recognizing characters such as letters, numerical charactersand symbols with accuracy even from an unclear image or an imageincluding noises, and an apparatus for the same method.

BACKGROUND ART

[0002] In the past, it has been performed to recognize letters in anobject image by comparing the object image with a reference image storedin a memory. For example, as disclosed in Japanese Patent Application[kokai] No. 8-212290, there is a method of identifying letters includedin an original image, which comprises the steps of binarizing theletters to be identified, performing a normalization treatment to theobtained binary image, and inputting the normalized data into a neuralnetwork. According to this method, it is possible to accurately identifyletters and/or numerical characters even from a number plate of a movingautomobile.

[0003] However, in this method, there is a case that accuraterecognition results can not be obtained when the original image includessome noises and/or blur. In particular, when a quality of the originalimage is relatively low, for example, the original image includes somecharacters such as numerical characters, each of which is composed of aplurality of elements, and/or undesired dots around the characters inthe background, as shown in FIG. 1A, there is a fear that time needed torecognize the characters considerably increases despite a decrease inrecognition accuracy.

SUMMARY OF THE INVENTION

[0004] Therefore, a concern of the present invention is to provide animage recognition method including the capability of efficientlyrecognizing characters such as letters, numerical characters and symbolsincluded in an original image with accuracy even when at least one ofthe characters included in the original image is composed of a pluralityof elements.

[0005] That is, the image recognition method of the present inventioncomprises the steps of:

[0006] (I) taking a first image including a character composed of pluralelements;

[0007] (II) extracting the plural elements in the first image to obtaina second image, in which each of the plural elements is enclosed by arectangular frame;

[0008] (III) forming a composite image with respect to a combination ofthe rectangular frames in the second image;

[0009] (IV) calculating a characteristic amount of the elements includedin the composite image;

[0010] (V) inputting the characteristic amount in a back-propagationnetwork, in which learning about a reference character(s) to be includedin the first image has already finished, to provide a degree ofagreement between the characteristic amount of the composite image andthe reference character; and

[0011] (VI) determining the composite image having a highest degree ofagreement between the characteristic amount of the composite image andthe reference character from results obtained by repeating the steps(III) to (V) with respect to different combinations of the rectangularframes in the second image, to output it as a recognition data.

[0012] In the above image recognition method, it is preferred to carryout a pretreatment described below when the first image includes atleast two characters coupled to each other. That is, this pretreatmentcomprises the steps of:

[0013] preparing a binary image including the at least two characters;

[0014] determining a profile showing a distribution strength in analignment direction of the at least two characters from the binaryimage;

[0015] setting a threshold line having a predetermined distributionstrength in the profile;

[0016] removing a first region of the profile, where the distributionstrength is lower than the threshold line, from the profile to obtain acompensated image, in which the at least two characters are separatedfrom each other; and

[0017] using the compensated image as the first image.

[0018] In particular, it is preferred that the pretreatment describedabove comprises the steps of:

[0019] after removing the first region from the profile, dividing thefirst region into two areas at a position having a minimum distributionstrength of the profile within the first region; and

[0020] adding the two areas respectively into a pair of second regionsof the profile located at both sides of the first region, where thedistribution strength is greater than the threshold line, to obtain thecompensated image.

[0021] In addition, in the above image recognition method, it ispreferred to carry out a pretreatment described below when the firstimage includes a character composed of a plurality of dots. That is,this pretreatment comprises the steps of:

[0022] preparing a binary image including the character of the dots;

[0023] expanding each of the dots of the character in a horizontaldirection in the binary image to obtain a compensated image, in whicheach of expanded dots are joined with an adjacent expanded dot; and

[0024] using the compensated image as the first image.

[0025] In particular, it is preferred that this pretreatment comprisesthe steps of:

[0026] preparing a binary image including the character of the dots;

[0027] expanding each of the dots of the character in horizontal andvertical directions in the binary image to obtain a compensated image,in which each of expanded dots are joined with an adjacent expanded dot;and

[0028] using the compensated image as the first image.

[0029] Another concern of the present invention is to provide an imagerecognition apparatus including the capability of achieving theremarkable effects of the above-described image recognition method.

[0030] The image recognition apparatus of the present inventioncomprises:

[0031] an image pickup device for taking a first image;

[0032] an image-element divider for extracting a plurality of elementsconstructing a character included in the first image to obtain a secondimage, in which each of the elements is enclosed by a rectangular frame;

[0033] a composite-image generator for forming a composite image withrespect to a combination of the rectangular frames in the second image;

[0034] a characteristic-amount calculator for determining acharacteristic amount of the elements included in-the composite image;

[0035] a back-propagation network, in which learning about a referencecharacter(s) to be included in the first image has already finished, forproviding a degree of agreement between the characteristic amount of thecomposite image and the reference character when the characteristicamount is input into the back-propagation network; and

[0036] an image analyzer for determining the composite image having ahighest degree of agreement between the characteristic amount of thecomposite image and the reference character from results provided by theback-propagation network with respect to different combinations of therectangular frames in the second image, to output it as a recognitiondata.

[0037] These and still other objects and advantages of the presentinvention will become apparent from the best mode for carrying out theinvention explained below referring to the attached drawings.

BRIEF EXPLANATION OF THE DRAWINGS

[0038]FIGS. 1A to 1D are images showing an image recognition methodaccording to a first embodiment of the present invention;

[0039]FIG. 2 is a schematic diagram illustrating arrangements ofrectangular inner frames in a dotted circle of FIG. 1B;

[0040]FIGS. 3A and 3B are schematic diagrams illustrating examples ofrectangular outer frames;

[0041]FIG. 4 is an image showing an example of a first composite imageprovided via an area check step (II);

[0042]FIG. 5 is a flow chart of a first stage of the image recognitionmethod of the first embodiment;

[0043]FIGS. 6A and 6B are schematic diagrams illustrating examples of asecond stage of the image recognition method of the first embodiment;

[0044]FIG. 7 is a flow chart of the second stage of the imagerecognition method;

[0045]FIG. 8 is a flow chart of a third stage of the image recognitionmethod of the first embodiment;

[0046]FIG. 9 is an image showing an example of the first composite imageprovided via the second stage of the image recognition method;

[0047]FIGS. 10A and 10B are schematic diagrams illustrating examples ofsecond composite images;

[0048]FIGS. 11A to 11F are schematic diagrams illustrating preparationof data for neural network from the second composite image;

[0049]FIG. 12 is an image showing an example of recognition results ofthe image recognizing method of this embodiment;

[0050]FIGS. 13A to 13E are images showing a pretreatment performed priorto the image recognition method according to a second embodiment of thepresent invention;

[0051]FIGS. 14A and 14B are schematic diagrams illustrating thispretreatment;

[0052]FIG. 15 is an original image having a poor quality;

[0053]FIGS. 16A to 16D are images showing a pretreatment performed priorto the image recognition method according to a third embodiment of thepresent invention; and

[0054]FIGS. 17A to 17D are images showing a pretreatment performed priorto the image recognition method according to a third embodiment of thepresent invention.

BEST MODE FOR CARRYING OUT THE INVENTION

[0055] <First Embodiment>

[0056] As a preferred embodiment of the image recognition method of thepresent invention, a method of efficiently recognizing an arrangement ofnumerical characters with accuracy from an original image shown in FIG.1A, which is obtained by use of an image pick-up unit such as a TVcamera or a digital camera, is explained in detail below.

[0057] In the original image of FIG. 1A, for example, the numericalcharacter “1” is composed of upper and lower elements (11, 12) because acenter portion of the numerical character “1” has been lost. Similarly,the numerical character “2” is composed of four elements (21, 22, 23,24), and there are undesired dots (25, 26) as noises at upper and lowersides of the numerical character “2”.

[0058] In the image recognition method of this embodiment, each of theelements included in the original image, that is, each of the elements(11, 12, 21, 22, . . . ) constructing the respective numericalcharacters (“1”, “2”, . . . ) and the undesired dots (25, 26, . . . )included as the noises in the original image, is extracted, and thenenclosed by a rectangular inner frame 30 to obtain a second image shownin FIGS. 1B and 2. That is, in the second image, each of the rectangularframes 30 is arranged so as to circumscribe the respective element ordot. The second image can be obtained by a frame distributing unit forextracting the plurality of elements constructing the character in theoriginal image and arranging the rectangular inner frames 30 such thateach of the rectangular inner frames circumscribes the respectiveelement.

[0059] Next, a combination of the rectangular inner frames 30 in thesecond image is voluntarily selected, and a first composite image 100 isprepared by a frame synthesizer according to this combination. Forexample, the first composite image 100 can be prepared with respect totwo rectangular frames 30 enclosing the elements 11, 12 of the numericalcharacter “1” therein, as shown in FIG. 3A, and another first compositeimage 100 can be prepared with respect to five rectangular frames 30enclosing the elements 21 to 24 of the numerical character “2” and theundesired dot 24 therein, as shown in FIG. 3B. Thus, this firstcomposite image 100 is defined by a rectangular outer frame 40, whichcircumscribes a plurality of the rectangular inner frames 30. In thesecond image, the X-axis is defined in a direction of arrangement of thenumerical characters, and the Y-axis is defined in a height direction ofthe numerical characters.

[0060] Next, an area of the rectangular outer frame 40 of this firstcomposite image 100 is calculated. For example, as shown in FIG. 3A, thearea of the rectangular outer frame 40 can be easily calculatedaccording to coordinates of the upper left corner (x1, y1) and the lowerright corner (x2, y2) of the rectangular outer frame. Then, thecalculated area of the rectangular outer frame 40 is compared with arequired value previously stored in a back propagation network, forexample, 2 times an average of widths of numerical characters stored inthe back propagation network. When the area is equal to or smaller thanthe required value, the image recognition method proceeds to the nextstep. On the other hand, when the area is larger than the requiredvalue, the first composite image is deleted, and another combination ofthe rectangular inner frames 30 in the second image is selected togenerate another first composite image 100. The area of the rectangularouter frame of another first composite image is checked according to theabove-described manner. In this embodiment, this step is called as anarea check step (I).

[0061] After the area check step (I), a distance between the rectangularinner frames in the first composite image is calculated. For example, asshown in FIG. 3, this distance d can be easily determined according tocoordinates of the upper left corners (x1, y1), (x3, y3) of the tworectangular inner frames 30. Then, the calculated distance is comparedwith a required value previously stored in the back propagation network,for example, 40% of an average of gap widths of the numerical charactersstored in the back propagation network. When the distance is equal to orsmaller than the required value, the image recognition method proceedsto the next step. On the other hand, when the distance is larger thanthe required value, the first composite image 100 is deleted, andanother combination of the rectangular inner frames in the second imageis selected to generate another first composite image. The distancebetween the rectangular inner frames in another first composite image ischecked according to the above-described manner. In this embodiment,this step is called as a distance check step (I).

[0062] After the distance check step (I), the area of the rectangularouter frame 40 of the first composite image 100 is compared with arequired value previously stored in the back propagation network, forexample, a half of an average of widths of numerical characters_storedin the back propagation network. When the area is equal to or largerthan the required value, the image recognition method proceeds to thenext step. On the other hand, when the area is larger than the requiredvalue, another rectangular inner frame in the second image is selected,and added to the first composite image, so that another first compositeimage having an increased area is generated. An area of the rectangularouter frame 100 of another first composite image is checked according tothe above-described manner. Thus, by repeating this procedure until theabove condition is satisfied, a plurality of rectangular inner framescan be enclosed in a single rectangular outer frame, as shown in FIG.3B. In this embodiment, this step is called as an area check step (II).

[0063] As shown in FIG. 1C, the first composite image 100 provided viathe area check step (II) is stored as a reliable candidate image in amemory. An example of the first composite image 100 provided via thearea check step (II) is shown in FIG. 4. In this figure, although aplurality of the elements and the dots are included in the firstcomposite image 100, it should be noted that all of the elementsconstructing the numerical character “3” are included in the firstcomposite image 100. However, there is a case that all of the elementsconstructing the numerical character are not included in the firstcomposite image. Therefore, if such a useless first composite image canbe found and deleted prior to the subsequent important steps of theimage recognition method of the present invention, it is possible toreduce a total number of the first composite images in order to moreefficiently carry out the image recognition. Therefore, a second stageof the image recognition method of this embodiment is an optional stagefor achieving this purpose, i.e., “data reduction”. Therefore, thesecond stage may be omitted, if necessary.

[0064] Prior to the explanation of the second stage, the first stageexplained above of the image recognition method of this embodiment issummarized according to a flow chart shown in FIG. 5. That is, each ofthe rectangular inner frames 30 that circumscribes the respectiveelement or dot in the second image is selected as a combination baseelement in order (step 50). In addition, another rectangular inner frame30 to be combined with the combination base element is selected (step51), to thereby prepare the first composite image 100 having therectangular outer frame 40 that circumscribes those selected rectangularinner frames 30 (step 52).

[0065] Then, the area of the rectangular outer frame 40 of the firstcomposite image 100 is calculated (step 53), and the area-check step(I), the distance-check step (I) and the area-check step (II) areperformed in order (steps 54-56). When the first composite image 100 isregarded as “No Good (NG)” in either the area-check step (I) or thedistance-check step (I), it is deleted, and a new rectangular innerframe to be combined with the rectangular inner frame of the combinationbase element is selected (step 51), to thereby generate another firstcomposite image. On the other hand, when the first composite image 100is regarded as “No Good (NG)” in the area-check step (II), an additionalrectangular inner frame 30 is selected and added to the first compositeimage 30 to increase the total area thereof. Therefore, even if thefirst composite image 100 is regarded as “NG” in the area-check step(II), it is not deleted.

[0066] As described above, the first composite image 100 (e.g., FIG. 4)provided via all of the area-check step (I), the distance-check step (I)and the area-check step (II) is stored as the reliable candidate imagein the memory (step 57). By the way, when the first composite image 100is stored as the reliable data in the memory, a next first compositeimage is prepared by combining another rectangular inner frame selectedas a new combination base element with at least one of the remainingrectangular inner frames other than the rectangular inner frame(s) thathas already used as the combination base element. According to thismanner, when all of the rectangular inner frames in the second image areused as the combination base element (step 58), the image recognitionmethod of the present invention proceeds to the second stage that is thedata reduction treatment, as shown in a flow chart of FIG. 6.

[0067] The second stage of the image recognition method of thisembodiment is a data reduction treatment of deleting overlapping firstcomposite images to achieve an improvement in the recognition speed.That is, as shown in the flow chart of FIG. 7, one of the firstcomposite images stored in the memory is selected, and another firstcomposite image to be compared with the selected one of the firstcomposite image (step 60), so that a comparison therebetween isperformed. For example, as shown in FIG. 6A, when the first compositeimage 100′ is completely enclosed in the other first composite image 100(step 60), the first composite image 100′ is deleted (step 62).

[0068] On the other hand, as shown in FIG. 6B, when the first compositeimage 100 is partially overlapped with the other first composite image100′, an overlapping area between those first composite images (100,100′) is calculated. When the overlapping area is 80% or more of one ofthe first composite images (step 63), the first composite images (100,100′) are coupled to each other to generate a new first composite image100″ (step 64). When the overlapping area is less than 80%, each of thefirst composite images is maintained on one's own without being coupledwith the other first composite image. When all of the first compositeimages in the memory have been checked in the second stage (step 65),the image recognition method of this embodiment proceeds to the thirdstage.

[0069] In the third stage of the image recognition method, as shown in aflow chart of FIG. 8, steps similar to the area check step (I), thedistance check step (I) and the area check step (II) of the first stageare substantially repeated with respect to a plurality of therectangular inner frames 30 included in the rectangular outer frame 40of the first composite image 100 provided from the second stage. Inother words, the first stage is to determine a coarse (wide) region(=first composite image) including the elements of the numericalcharacter to be recognized, and on the contrary the third stage is todetermine a fine (narrow) region (=second composite image describedlater)) substantially including only the elements of the numericalcharacter to be recognized, and achieve the image recognition accordingto the second composite image.

[0070] In the third stage, as shown in FIGS. 9 and 10A, a combination ofthe rectangular inner frames 30 in the first composite image 100 isvoluntarily selected, and a second composite image 200 is preparedaccording to this combination by the substantially same manner as thefirst stage. This second composite image 200 is defined by a rectangularouter frame 70, which circumscribes the selected rectangular innerframes 30.

[0071] Next, with respect to this second composite image 200, an area ofthe rectangular outer frame 70 is calculated, for example by thesubstantially same manner as the first stage. The calculated area of therectangular outer frame 70 is compared with a required value previouslystored in the back propagation network, for example, 1.2 times anaverage of widths of the numerical characters stored in the backpropagation network. In the third stage, this comparison is performedunder a more severe condition than the first stage (e.g., 2 times theaverage of widths of the numerical characters stored in the backpropagation network). When the area is equal to or smaller than therequired value, the image recognition method proceeds to the next stepof the third stage. On the other hand, when the area is larger than therequired value, the second composite image is deleted, and anothercombination of the rectangular inner frames 30 in the first compositeimage 100 is selected to generate another second composite image 200.The area of the rectangular outer frame 70 of another second compositeimage is checked according to the above-described manner. In thisembodiment, this step is called as an area check step (III).

[0072] After the area check step (III), a distance between therectangular inner frames 30 in the second composite image 200 iscalculated, for example, by the substantially same manner as the firststage. Then, the calculated distance is compared with a required valuepreviously stored in the back propagation network, for example, 40% ofan average of gap widths of the numerical characters stored in the backpropagation network. When the distance is equal to or smaller than therequired value, the image recognition method proceeds to the next stepof the third image. On the other hand, when the distance is larger thanthe required value, the second composite image 200 is deleted, andanother combination of the rectangular inner frames 30 in the firstcomposite image 100 is selected to generate another second compositeimage 200. The distance between the rectangular inner frames 30 inanother second composite image 200 is checked according to theabove-described manner. In this embodiment, this step is called as adistance check step (II).

[0073] After the distance check step (II), the area of the rectangularouter frame 70 of this second composite image 200 is compared with arequired value previously stored in the back propagation network, forexample, 0.8 times an average of widths of the numerical charactersstored in the back propagation network. In the third stage, thecomparison is performed under a more severe condition than the firststage (e.g., a half of an average of widths of the numerical charactersstored in the back propagation network). When the area is equal to orlarger than the required value, the image recognition method of thepresent invention proceeds to the next step of the third stage. On theother hand, when the area is larger than the required value, anotherrectangular inner frame 30 in the first composite image 100 is selected,and added to the second composite image, so that a new second compositeimage having an increased area is generated. An area of the rectangularouter frame 70 of the new second composite image 200 is checkedaccording to the above-described manner. Thus, by repeating thisprocedure until the above condition is satisfied, a plurality ofrectangular inner frames 30 in the first composite image 100 can beenclosed in a single rectangular outer frame 70 of the second compositeimage 200, as shown in FIG. 10B. In this embodiment, this step is calledas an area check step (IV).

[0074] The flow chart diagram shown in FIG. 8 summarizes the third stageof the image recognition method of this embodiment so far. That is, eachof the rectangular inner frames 30 enclosing the elements or dots in thefirst composite image 100 is selected as a combination base element inorder (step 80). In addition, another rectangular inner frame 30 to becombined with the rectangular inner frame of the combination baseelement is selected (step 81), to thereby obtain the second compositeimage 200 having the rectangular outer frame 70 that circumscribes thoseselected rectangular inner frames 30 (step 82), as shown in FIG. 10A.

[0075] Then, the area of the rectangular outer frame 70 of the secondcomposite image 200 is calculated (step 83), and the area-check step(III), the distance-check step (II) and the area-check step (IV) arecarried out in order (steps 85-87). When the second composite image 200is regarded as “No Good (NG)” in either the area-check step (III) or thedistance-check step (II), it is deleted, and another rectangular innerframe 30 to be combined with the rectangular inner frame of thecombination base element is selected (step 81), to thereby generateanother second composite image 200 (step 82). On the other hand, whenthe second composite image 200 is regarded as “No Good (NG)” in thearea-check step (IV), an additional rectangular inner frame 30 isselected, and added to the second composite image regarded as “NG”, sothat a new second composite image 200 having an increased area isgenerated. Therefore, the second composite image regarded as “NG” in theare-check step (II) is not deleted. By the way, the third stagecomprises a step 84 of checking as to whether the subsequenttreatment(s), e.g., the area-check step (III) has been already performedto the second composite image 200. If yes, the second composite image isdeleted to avoid unnecessarily repeating the same treatment.

[0076] Next, an aspect ratio, i.e., vertical width/horizontal width ofthe rectangular outer frame 70 of the second composite image 200provided via the area-check step (IV) is calculated, and compared with apredetermined value stored in the back propagation network (step 88). Inthis embodiment, this step that is called as a aspect-ratio check step(I) is performed as to whether the aspect ration is within a range of0.5 times to 1.5 times a minimum value of the aspect ratios of thenumerical characters stored in the back propagation network. When theaspect ratio is within this range, the image recognition method proceedsto the next step of the third stage. When the aspect ratio is out ofthis range, the second composite image is deleted, and anotherrectangular inner frame 30 in the first composite image 100 is selectedto generate a new second composite image 200.

[0077] Next, data to be input into the neural network is prepared (step90) from the second composite image 200 regarded as “OK” in theaspect-ratio check step (I). First, the elements included in the secondcomposite image 200 are extracted. In this embodiment, for example, asshown in FIG. 11A, the two elements constructing the numerical character“4” are extracted from the second composite image 200. Next, an imagesize of the extracted image is normalized. For example, thenormalization is performed such that a longer one (“Ly” in FIG. 11A) ofthe X-axis and Y-axis lengths (Lx, Ly) of the extracted image is 40pixels.

[0078] Next, with respect to each of the pixels on edge lines of theelements of the normalized image, an outline direction is calculated. Inthis embodiment, as shown in FIG. 11B, four outline directions (D1 toD4) are set. For example, in the region indicated by a circle in FIG.11B, when the outline direction of a target pixel Pt is defined as adirection of a straight line extending between the target pixel Pt and areference pixel Ps positioned backward from the target pixel Pt by only1 pixel, it is regarded that the target pixel Pt has the outlinedirection D2 that is an oblique direction. However, this resultdisaccords with the fact that a correct outline direction of the targetpixel Pt should be regarded as D3. In this embodiment, since the outlinedirection of the target pixel Pt is defined as a direction of a straightline extending between the target pixel Pt and a reference pixel Ps'positioned backward from the target pixel by 6 pixels, it is regardedthat the target pixel Pt has the outline direction D3 that is a verticaldirection. This result accords with the fact that the correct outlinedirection is D3, as described above.

[0079] Next, as shown in FIG. 11E, a grid pattern 45 is placed such thatthe elements of the normalized image are included therein. In thisembodiment, this grid pattern 45 is configured in a square shape of40×40 pixels. The number of grids of the grid pattern 45 is 36. Withrespect to the pixels on the edge line included in each of the grids, ahistogram of the outline directions is prepared, as shown in FIG. 11F.After a treatment of dividing a height of each of the outline directionsby the maximum height of the outline direction is performed with respectto each of the histograms, resultant data are input in the neuralnetwork (step 91). In this embodiment, input dimensions of the neuralnetwork are 144, which is obtained by multiplying the number of outlinedirections (=4 dimensions) by the number of grids (=36 dimensions). Thecalculation of the neural network presents a recognition result thenumerical character included in the second composite image 200 is “4”(step 92).

[0080] To obtain the recognition result with the highest accuracy, theaspect ratio of the recognized numerical character (for example, “4”) iscalculated, and compared with a reference aspect ratio (for example, thereference aspect ratio of “4”) stored in the back propagation network tocheck a degree of agreement therebetween (step 93). In this embodiment,this is called as an aspect-ratio check step (II). When a recognitionresult with a degree of agreement therebetween is obtained, it istemporarily stored in the memory. Subsequently, if a new recognitionresult having a higher degree of agreement therebetween is obtained, thedata is renewed (steps 94 and 95). By repeating this procedure, therecognition result having the highest degree of agreement therebetweencan be obtained. In this embodiment, for example, as shown in FIG. 12,the combination of the two elements designated by the numeral 4 has ahigher degree of agreement of the aspect ratio than any combinationincluding the dot(s) designated “X”. Therefore, the combination of therectangular inner frames 30 with the highest degree of agreement isoutput as a correct combination of the elements constructing thenumerical character “4”.

[0081] The step 96 of the third stage is to check as to whether all ofthe rectangular inner frames 30 in the first composite image 100provided from the second stage have been used as the combination baseelement in the third stage. If yes, the image recognition method of thisembodiment proceeds to the next step 97, which is to check as to whetherthe recognition of all of the numerical characters in the original imageof FIG. 1A has been finished. As a result, as shown in FIG. 1D, theimage recognition method of the present invention can provide an imageincluding the rectangular outer frames 70 of the second composite images200, in each of which only the elements constructing the numericalcharacter are enclosed.

[0082] By the way, in this embodiment, the characteristic amounts suchas area, distance and aspect ratio can be determined by characteristicamount calculating units. In addition, the degree of agreement of thecharacteristic amount can be determined by an image analyzer. Therefore,the present embodiment also provides an image recognition apparatus orsystem for realizing the image recognition method described above.

[0083] <Second Embodiment>

[0084] This embodiment presents a pretreatment that is preferablyperformed prior to the image recognition method of the first embodiment,for example, when an original image to be recognized includes anarrangement of numerical characters, in which adjacent numericalcharacters are partially coupled to each other, as shown in FIG. 13A,because the original image is prepared under a bad condition. Therefore,a duplicate explanation of the image recognition method of the firstembodiment performed after the pretreatment is omitted.

[0085] First, as shown in FIG. 13B, a binary image of the original imageof FIG. 13A is prepared. Then, a profile indicative of distributionstrength in an alignment direction (x-axis direction) of the numericalcharacters from the binary image is determined, as shown in FIG. 13C.FIG. 13D is a top view of the profile of FIG. 13C, in which each ofbright regions designates a high distribution strength of the numericalcharacter. As the region becomes dark, it means that the distributionstrength of the numerical character is smaller, or zero (=the regionbetween adjacent numerical characters).

[0086] As an example, a method of determining the profile is introduced.First, the numerical character regions in the binary image (FIG. 13B)are projected on an axis that called as a projection axis (=x axis),extending parallel to the alignment direction of the numericalcharacters. In this projection treatment, scanning is performed in adirection (Y-axis) perpendicular to the projection axis to count thenumber of pixels on the scanning line. For example, as shown in FIG.13B, when the pixels of white regions indicative of the numericalcharacters provide a concentration value “1”, and the pixels of blackregions indicative of the background provide a concentration value “0”,the number of pixels having the concentration value “1” is counted. Bydetermining, as a projection value, the number of pixels with theconcentration value “1” on the scanning line extending from each pointof the projection axis, the profile of distribution strength can beobtained, as shown in FIG. 13C.

[0087] By the way, when adjacent numerical characters (e.g., “9” and“0”) are coupled to each other by a relatively large area, as shown inFIG. 13B, it is needed to distinguish the white region of the couplingregion from the white regions constructing the numerical character. Inthis method, by multiplexing the projection value by the number ofisland regions on the scanning line, in each of which the pixels havingthe concentration value “1” are successively arranged, it is possible todistinguish the pixels having the concentration value “1” of thenumerical character from the pixels having the concentration value “1”of the coupling region. For example, in FIG. 13B, the number of pixels“1” on the scanning line Ls1 passing the coupling region isapproximately equal to the number of pixels “1” on the scanning line Ls2passing the numerical character “0”. However, by performing themultiplexing treatment described above, the distribution strength at thescanning line Ls1 becomes lower than the distribution strength at thescanning line Ls2, as shown by square regions “Q1”, “Q2” in FIG. 13C.

[0088] Next, as shown in FIG. 13C, a threshold line L having apredetermined distribution strength is set in the profile. For example,when a lower region S2 of the profile where the distribution strength islower than the threshold line L, is positioned between a pair of upperregions (S1L, S1R) of the profile where the distribution strength isgreater than the threshold line L, the lower region S2 is divided intothe two areas (S2L, S2R) at a position “Pm” having a minimumdistribution strength of the profile within the lower region. FIG. 13Eis a bottom view of the profile of FIG. 13C, in which each of whitelines extending in the Y-axis direction designates the position “Pm” ofthe minimum distribution strength in the respective lower region S2 ofthe profile.

[0089] As shown in FIGS. 14A and 14B, these area (S2L, S2R) are removedfrom the profile, and then respectively added to the adjacent upperregions (S1L, S1R), so that the upper region S1L′ is separated from theadjacent upper region S1R′. Thus, a compensated image is obtained, inwhich the adjacent numerical characters of the original image areseparated from each other. By carrying out the image recognition methodof the first embodiment to this compensated image, the rectangular innerframes can be placed such that each of the rectangular frames enclosestherein a single numerical character. Therefore, it is possible to avoidthe occurrence of an inconvenience that the adjacent numericalcharacters coupled to each other are enclosed in a single rectangularframe, which may be a cause of lowering the recognition accuracy.

[0090] The threshold line L can be determined as follows. That is, anupper region of the profile is firstly extracted in the case that thethreshold line is set to a position “0”. Then, an aspect ratio of thisupper region is compared with a predetermined value, for example, anaverage aspect ratio of the numerical characters previously stored inthe back propagation network. When the aspect ratio of the upper regionis larger than the average aspect ratio, the threshold line is used.However, if this condition is not satisfied, the above procedures arerepeated by changing the position of the threshold line to determine thethreshold line satisfying the above condition. As a modification, anaverage width in the horizontal direction of the characters may be usedin place of the average aspect ratio.

[0091] Thus, according to the image recognition method of the presentinvention with the pretreatment of the second embodiment, it is possibleto efficiently recognize the characters such as numerical characters andletters with accuracy even from a bad-quality original image, forexample, as shown in FG. 15, in which some of the characters are dividedinto plural elements (e.g., “8” and “9”), undesired dots exist aroundthe characters, and some of the characters are coupled to each other(e.g., “S” and “H”).

[0092] <Third Embodiment>

[0093] This embodiment presents a pretreatment that is preferablyperformed prior to the image recognition method of the first embodiment,for example, when an original image to be recognized includes anarrangement of characters such as numerical characters and letters, eachof which is composed of a plurality of dots, as shown in FIG. 16A.Therefore, a duplicate explanation of the image recognition method ofthe first embodiment performed after the pretreatment is omitted.

[0094] In this pretreatment, as shown in FIG. 16B, a binary image of theoriginal image of FIG. 16A is firstly prepared. Then, each of the dotsof the characters is expanded in two directions, i.e., horizontal andvertical directions of the binary image to obtain a compensated image,as shown in FIG. 16C, in which each of the expanded dots are joined withan adjacent expanded dot. This expansion treatment can be performed byreplacing pixels having the value of “0” around the respective dot withthe pixels having the value of “1” in the designated direction of thebinary image. In the thus obtained compensated image, each of thecharacters is composed of a single element.

[0095] When the image recognition method of the present invention isperformed to the original image of FIG. 16A, there is a fear that timeneeded for the image recognition is extended because the rectangularinner frame is arranged with respect to each of the dots of thecharacters, so that the total number of the rectangular inner framesconsiderably increases. In addition, this may become a cause ofdeterioration in the recognition accuracy. However, when the imagerecognition method of the present invention is performed to thecompensated image of FIG. 16C, the total number of rectangular innerframes to be arranged can be remarkably reduced, as shown in FIG. 16D.Therefore, it is possible to achieve an improvement in recognitionaccuracy and a time saving for the image recognition. By the way, thereis a case that adjacent characters are coupled to each other in thiscompensated image. In such a case, the pretreatment explained in thesecond embodiment may be performed to the compensated image of FIG. 16C.

[0096] <Fourth Embodiment>

[0097] This embodiment presents a pretreatment that is preferablyperformed prior to the image recognition method of the first embodiment,for example, when an coupling area between adjacent characters in anoriginal image, as shown in FIG. 17A, is much larger than the case ofthe original image of FIG. 13A, so that a sufficient recognitionaccuracy may not be obtained by the pretreatment of the secondembodiment. Therefore, a duplicate explanation of the image recognitionmethod of the first embodiment performed after the pretreatment isomitted.

[0098] In this embodiment, when rectangular inner frames are arranged inan original image shown in FIG. 17A, two adjacent characters (“2” and“3” in FIG. 17A) coupled to each other are enclosed in a single largerectangular inner frame. Then, this large rectangular inner frame isforcedly divided into a plurality of regions each having a predeterminedarea to obtain a compensated image. For example, the large rectangularinner frame is divided into a plurality of small rectangular innerframes such that each of the small rectangular inner frames has a sidethat is substantially equal to a half of the minimum width in thehorizontal and vertical directions of the characters previously storedin the back propagation network. In FIG. 17C, the single largerectangular inner frame is divided into 16 (=4×4) small rectangularinner frames.

[0099] By carrying out the image recognition method of the firstembodiment to the obtained compensated image, it is possible toefficiently recognize the characters such as numerical characters andletters with accuracy, as shown in FIG. 17D, even from a bad-qualityoriginal image (e.g., FIG. 17A), in which the coupling area between theadjacent characters are relatively large. By the way, when the area ofthe large rectangular inner frame is smaller than a predetermined value,for example, 1.2 times an average area of the characters previouslystored in the back propagation network, it is preferred to carry out theimage recognition treatment of the present invention without thispretreatment.

INDUSTRIAL APPLICABILITY

[0100] According to the present invention, even when an original imagecan not be accurately recognized by a conventional binarization orprojection treatment because the original image includes characters suchas letters and numerical characters each composed of a plurality ofelements, or undesired dots around the characters, it is possible toprovide a reliable image recognition method. In addition, there is anadvantage of eliminating a problem that the accuracy of imagerecognition lowers, for example, when the characters to be recognizedhas an underline, or it is difficult to separate the characters from thebackground because the printing surface is a satin finished surface.Moreover, there is a further advantage of improving an inconveniencethat recognition results with accuracy can not be efficiently obtainedwith recognition accuracy when the original image includes some noisesin the background and/or characters each composed of a plurality ofdots, e.g., letters printed by an ink-jet printer.

[0101] Thus, since the image recognition method of the present inventioncan efficiently recognize the characters with accuracy even from anoriginal image having a poor quality, its applications are widelyexpected.

1. An image recognition method comprising the steps of: (I) taking afirst image including a character composed of plural elements; (II)extracting said plural elements in said first image to obtain a secondimage, in which each of said plural elements is enclosed by arectangular frame; (III) forming a composite image with respect to acombination of said rectangular frames in said second image; (IV)calculating a characteristic amount of said elements included in saidcomposite image; (V) inputting said characteristic amount in aback-propagation network, in which learning about a referencecharacter(s) to be included in said first image has already finished, toprovide a degree of agreement between the characteristic amount of saidcomposite image and the reference character; and (VI) determining saidcomposite image having a highest degree of agreement between thecharacteristic amount of said composite image and the referencecharacter from results obtained by repeating the steps (III) to (V) withrespect to different combinations of said rectangular frames in saidsecond image, to output it as a recognition data.
 2. The imagerecognition method as set forth in claim 1 comprising a pretreatmentperformed when said first image includes at least two characters coupledto each other, wherein said pretreatment comprises the steps of:preparing a binary image including said at least two characters;determining a profile showing a distribution strength in an alignmentdirection of said at least two characters from said binary image;setting a threshold line having a predetermined distribution strength inthe profile; removing a first region of said profile, where thedistribution strength is lower than the threshold line, from saidprofile to obtain a compensated image, in which said at least twocharacters are separated from each other; and using the compensatedimage as said first image.
 3. The image recognition method as set forthin claim 2, wherein said pretreatment comprises the steps of: afterremoving the first region from said profile, dividing the first regioninto two areas at a position having a minimum distribution strength ofsaid profile within the first region; and adding said two areasrespectively into a pair of second regions of said profile located atboth sides of said first region, where the distribution strength isgreater than the threshold line, to obtain said compensated image. 4.The image recognition method as set forth in claim 1 comprising apretreatment performed when said first image includes a charactercomposed of a plurality of dots, wherein said pretreatment comprises thesteps of: preparing a binary image including said character of the dots;expanding each of the dots of said character in a horizontal directionin said binary image to obtain a compensated image, in which each ofexpanded dots are joined with an adjacent expanded dot; and using thecompensated image as said first image.
 5. The image recognition methodas set forth in claim 1 comprising a pretreatment performed when saidfirst image includes a character composed of a plurality of dots,wherein said pretreatment comprises the steps of: preparing a binaryimage including said character of the dots; expanding each of the dotsof said character in horizontal and vertical directions in said binaryimage to obtain a compensated image, in which each of expanded dots arejoined with an adjacent expanded dot; and using the compensated image assaid first image.
 6. The image recognition method as set forth in claim1, wherein said characteristic amount is an aspect ratio of saidcomposite image.
 7. An image recognition apparatus comprising: an imagepickup device for taking a first image; an image-element divider forextracting a plurality of elements constructing a character included insaid first image to obtain a second image, in which each of saidelements is enclosed by a rectangular frame; a composite-image generatorfor forming a composite image with respect to a combination of saidrectangular frames in said second image; a characteristic-amountcalculator for determining a characteristic amount of said elementsincluded in said composite image; a back-propagation network, in whichlearning about a reference character(s) to be included in said firstimage has already finished, for providing a degree of agreement betweenthe characteristic amount of said composite image and the referencecharacter when the characteristic amount is input into saidback-propagation network; and an image analyzer for determining saidcomposite image having a highest degree of agreement between thecharacteristic amount of said composite image and the referencecharacter from results provided by said back-propagation network withrespect to different combinations of said rectangular frames in saidsecond image, to output it as a recognition data.